1 link tagged with all of: mechanistic interpretability + model internalization + qwen + gpt 2
Links
A researcher tested whether Qwen (a large language model) has internalized representations of GPT-2's text generation patterns by having Qwen complete text that GPT-2 started. Results suggest Qwen can mimic GPT-2's style and infer that GPT-2-generated text is from an earlier era, implying modern LLMs may develop internal models of other models during training.
- When Qwen completed GPT-2-generated text prefixes, its continuations overlapped more with GPT-2's actual completions than with Qwen's own natural completions of the same prompts, across multiple overlap metrics.
- Qwen inferred significantly earlier years (matching GPT-2's pre-2019 training data) when asked to complete years in GPT-2-generated articles versus its own generations, even after removing articles with explicit year references.
- The experiment used 12 internet headlines from September 2026 (outside both models' training cutoffs) with varying completion lengths (96-512 tokens), and results held up when tested on Qwen's 14B parameter version.
qwen
gpt 2
model internalization
mechanistic interpretability